Automatic detection of cell-cycle stages using recurrent neural networks.

Mitosis is the process by which eukaryotic cells divide to produce two similar daughter cells with identical genetic material. Research into the process of mitosis is therefore of critical importance both for the basic understanding of cell biology and for the clinical approach to manifold pathologies resulting from its malfunctioning, including cancer. In this paper, we propose an approach to study mitotic progression automatically using deep learning. We used neural networks to predict different mitosis stages. We extracted video sequences of cells undergoing division and trained a Recurrent Neural Network (RNN) to extract image features. The use of RNN enabled better extraction of features. The RNN-based approach gave better performance compared to classifier based feature extraction methods which do not use time information. Evaluation of precision, recall, and F-score indicates the superiority of the proposed model compared to the baseline. To study the loss in performance due to confusion between adjacent classes, we plotted the confusion matrix as well. In addition, we visualized the feature space to understand why RNNs are better at classifying the mitosis stages than other classifier models, which indicated the formation of strong clusters for the different classes, clearly confirming the advantage of the proposed RNN-based approach.

Denoising diffusion probabilistic models for generation of realistic fully-annotated microscopy image datasets.

Recent advances in computer vision have led to significant progress in the generation of realistic image data, with denoising diffusion probabilistic models proving to be a particularly effective method. In this study, we demonstrate that diffusion models can effectively generate fully-annotated microscopy image data sets through an unsupervised and intuitive approach, using rough sketches of desired structures as the starting point. The proposed pipeline helps to reduce the reliance on manual annotations when training deep learning-based segmentation approaches and enables the segmentation of diverse datasets without the need for human annotations. We demonstrate that segmentation models trained with a small set of synthetic image data reach accuracy levels comparable to those of generalist models trained with a large and diverse collection of manually annotated image data, thereby offering a streamlined and specialized application of segmentation models.

Surface-Patterned DNA Origami Rulers Reveal Nanoscale Distance Dependency of the Epidermal Growth Factor Receptor Activation.

The nanoscale arrangement of ligands can have a major effect on the activation of membrane receptor proteins and thus cellular communication mechanisms. Here we report on the technological development and use of tailored DNA origami-based molecular rulers to fabricate "Multiscale Origami Structures As Interface for Cells" (MOSAIC), to enable the systematic investigation of the effect of the nanoscale spacing of epidermal growth factor (EGF) ligands on the activation of the EGF receptor (EGFR). MOSAIC-based analyses revealed that EGF distances of about 30-40 nm led to the highest response in EGFR activation of adherent MCF7 and Hela cells. Our study emphasizes the significance of DNA-based platforms for the detailed investigation of the molecular mechanisms of cellular signaling cascades.

A Deep Learning-based in silico Framework for Optimization on Retinal Prosthetic Stimulation.

We propose a neural network-based framework to optimize the perceptions simulated by the in silico retinal implant model pulse2percept. The overall pipeline consists of a trainable encoder, a pre-trained retinal implant model and a pre-trained evaluator. The encoder is a U-Net, which takes the original image and outputs the stimulus. The pre-trained retinal implant model is also a U-Net, which is trained to mimic the biomimetic perceptual model implemented in pulse2percept. The evaluator is a shallow VGG classifier, which is trained with original images. Based on 10,000 test images from the MNIST dataset, we show that the convolutional neural network-based encoder performs significantly better than the trivial downsampling approach, yielding a boost in the weighted F1-Score by 36.17% in the pre-trained classifier with 6×10 electrodes. With this fully neural network-based encoder, the quality of the downstream perceptions can be fine-tuned using gradient descent in an end-to-end fashion.

Multimodal Deep Learning for Integrating Chest Radiographs and Clinical Parameters: A Case for Transformers.

Background Clinicians consider both imaging and nonimaging data when diagnosing diseases; however, current machine learning approaches primarily consider data from a single modality. Purpose To develop a neural network architecture capable of integrating multimodal patient data and compare its performance to models incorporating a single modality for diagnosing up to 25 pathologic conditions. Materials and Methods In this retrospective study, imaging and nonimaging patient data were extracted from the Medical Information Mart for Intensive Care (MIMIC) database and an internal database comprised of chest radiographs and clinical parameters inpatients in the intensive care unit (ICU) (January 2008 to December 2020). The MIMIC and internal data sets were each split into training (n = 33 893, n = 28 809), validation (n = 740, n = 7203), and test (n = 1909, n = 9004) sets. A novel transformer-based neural network architecture was trained to diagnose up to 25 conditions using nonimaging data alone, imaging data alone, or multimodal data. Diagnostic performance was assessed using area under the receiver operating characteristic curve (AUC) analysis. Results The MIMIC and internal data sets included 36 542 patients (mean age, 63 years ± 17 [SD]; 20 567 male patients) and 45 016 patients (mean age, 66 years ± 16; 27 577 male patients), respectively. The multimodal model showed improved diagnostic performance for all pathologic conditions. For the MIMIC data set, the mean AUC was 0.77 (95% CI: 0.77, 0.78) when both chest radiographs and clinical parameters were used, compared with 0.70 (95% CI: 0.69, 0.71; P < .001) for only chest radiographs and 0.72 (95% CI: 0.72, 0.73; P < .001) for only clinical parameters. These findings were confirmed on the internal data set. Conclusion A model trained on imaging and nonimaging data outperformed models trained on only one type of data for diagnosing multiple diseases in patients in an ICU setting. © RSNA, 2023 Supplemental material is available for this article. See also the editorial by Kitamura and Topol in this issue.

Medical transformer for multimodal survival prediction in intensive care: integration of imaging and non-imaging data.

When clinicians assess the prognosis of patients in intensive care, they take imaging and non-imaging data into account. In contrast, many traditional machine learning models rely on only one of these modalities, limiting their potential in medical applications. This work proposes and evaluates a transformer-based neural network as a novel AI architecture that integrates multimodal patient data, i.e., imaging data (chest radiographs) and non-imaging data (clinical data). We evaluate the performance of our model in a retrospective study with 6,125 patients in intensive care. We show that the combined model (area under the receiver operating characteristic curve [AUROC] of 0.863) is superior to the radiographs-only model (AUROC = 0.811, p < 0.001) and the clinical data-only model (AUROC = 0.785, p < 0.001) when tasked with predicting in-hospital survival per patient. Furthermore, we demonstrate that our proposed model is robust in cases where not all (clinical) data points are available.

Denoising diffusion probabilistic models for 3D medical image generation.

Recent advances in computer vision have shown promising results in image generation. Diffusion probabilistic models have generated realistic images from textual input, as demonstrated by DALL-E 2, Imagen, and Stable Diffusion. However, their use in medicine, where imaging data typically comprises three-dimensional volumes, has not been systematically evaluated. Synthetic images may play a crucial role in privacy-preserving artificial intelligence and can also be used to augment small datasets. We show that diffusion probabilistic models can synthesize high-quality medical data for magnetic resonance imaging (MRI) and computed tomography (CT). For quantitative evaluation, two radiologists rated the quality of the synthesized images regarding "realistic image appearance", "anatomical correctness", and "consistency between slices". Furthermore, we demonstrate that synthetic images can be used in self-supervised pre-training and improve the performance of breast segmentation models when data is scarce (Dice scores, 0.91 [without synthetic data], 0.95 [with synthetic data]).

A Deep-Learning-Computed Cancer Score for the Identification of Human Hepatocellular Carcinoma Area Based on a Six-Colour Multiplex Immunofluorescence Panel.

Liver cancer is one of the most frequently diagnosed and fatal cancers worldwide, with hepatocellular carcinoma (HCC) being the most common primary liver cancer. Hundreds of studies involving thousands of patients have now been analysed across different cancer types, including HCC, regarding the effects of immune infiltrates on the prognosis of cancer patients. However, for these analyses, an unambiguous delineation of the cancer area is paramount, which is difficult due to the strong heterogeneity and considerable inter-operator variability induced by qualitative visual assessment and manual assignment. Nowadays, however, multiplex analyses allow the simultaneous evaluation of multiple protein markers, which, in conjunction with recent machine learning approaches, may offer great potential for the objective, enhanced identification of cancer areas with further in situ analysis of prognostic immune parameters. In this study, we, therefore, used an exemplary five-marker multiplex immunofluorescence panel of commonly studied markers for prognosis (CD3 T, CD4 T helper, CD8 cytotoxic T, FoxP3 regulatory T, and PD-L1) and DAPI to assess which analytical approach is best suited to combine morphological and immunohistochemical data into a cancer score to identify the cancer area that best matches an independent pathologist's assignment. For each cell, a total of 68 individual cell features were determined, which were used as input for 4 different approaches for computing a cancer score: a correlation-based selection of individual cell features, a MANOVA-based selection of features, a multilayer perceptron, and a convolutional neural network (a U-net). Accuracy was used to evaluate performance. With a mean accuracy of 75%, the U-net was best capable of identifying the cancer area. Although individual cell features showed a strong heterogeneity between patients, the spatial representations obtained with the computed cancer scores delineate HCC well from non-cancer liver tissues. Future analyses with larger sample sizes will help to improve the model and enable direct, in-depth investigations of prognostic parameters, ultimately enabling precision medicine.

Definition and Quantification of Three-Dimensional Imaging Targets to Phenotype Pre-Eclampsia Subtypes: An Exploratory Study.

Pre-eclampsia is a severe placenta-related complication of pregnancy with limited early diagnostic and therapeutic options. Aetiological knowledge is controversial, and there is no universal consensus on what constitutes the early and late phenotypes of pre-eclampsia. Phenotyping of native placental three-dimensional (3D) morphology offers a novel approach to improve our understanding of the structural placental abnormalities in pre-eclampsia. Healthy and pre-eclamptic placental tissues were imaged with multiphoton microscopy (MPM). Imaging based on inherent signal (collagen, and cytoplasm) and fluorescent staining (nuclei, and blood vessels) enabled the visualization of placental villous tissue with subcellular resolution. Images were analysed with a combination of open source (FIJI, VMTK, Stardist, MATLAB, DBSCAN), and commercially (MATLAB) available software. Trophoblast organization, 3D-villous tree structure, syncytial knots, fibrosis, and 3D-vascular networks were identified as quantifiable imaging targets. Preliminary data indicate increased syncytial knot density with characteristic elongated shape, higher occurrence of paddle-like villous sprouts, abnormal villous volume-to-surface ratio, and decreased vascular density in pre-eclampsia compared to control placentas. The preliminary data presented indicate the potential of quantifying 3D microscopic images for identifying different morphological features and phenotyping pre-eclampsia in placental villous tissue.

Artificial Intelligence for Clinical Interpretation of Bedside Chest Radiographs.

Background Supine chest radiography for bedridden patients in intensive care units (ICUs) is one of the most frequently ordered imaging studies worldwide. Purpose To evaluate the diagnostic performance of a neural network-based model that is trained on structured semiquantitative radiologic reports of bedside chest radiographs. Materials and Methods For this retrospective single-center study, children and adults in the ICU of a university hospital who had been imaged using bedside chest radiography from January 2009 to December 2020 were reported by using a structured and itemized template. Ninety-eight radiologists rated the radiographs semiquantitatively for the severity of disease patterns. These data were used to train a neural network to identify cardiomegaly, pulmonary congestion, pleural effusion, pulmonary opacities, and atelectasis. A held-out internal test set (100 radiographs from 100 patients) that was assessed independently by an expert panel of six radiologists provided the ground truth. Individual assessments by each of these six radiologists, by two nonradiologist physicians in the ICU, and by the neural network were compared with the ground truth. Separately, the nonradiologist physicians assessed the images without and with preliminary readings provided by the neural network. The weighted Cohen κ coefficient was used to measure agreement between the readers and the ground truth. Results A total of 193 566 radiographs in 45 016 patients (mean age, 66 years ± 16 [SD]; 61% men) were included and divided into training (n = 122 294; 64%), validation (n = 31 243; 16%), and test (n = 40 029; 20%) sets. The neural network exhibited higher agreement with a majority vote of the expert panel (κ = 0.86) than each individual radiologist compared with the majority vote of the expert panel (κ = 0.81 to ≤0.84). When the neural network provided preliminary readings, the reports of the nonradiologist physicians improved considerably (aided vs unaided, κ = 0.87 vs 0.79, respectively; P < .001). Conclusion A neural network trained with structured semiquantitative bedside chest radiography reports allowed nonradiologist physicians improved interpretations compared with the consensus reading of expert radiologists. © RSNA, 2022 Supplemental material is available for this article. See also the editorial by Wielpütz in this issue.

LiveCellMiner: A new tool to analyze mitotic progression.

Live-cell imaging has become state of the art to accurately identify the nature of mitotic and cell cycle defects. Low- and high-throughput microscopy setups have yield huge data amounts of cells recorded in different experimental and pathological conditions. Tailored semi-automated and automated image analysis approaches allow the analysis of high-content screening data sets, saving time and avoiding bias. However, they were mostly designed for very specific experimental setups, which restricts their flexibility and usability. The general need for dedicated experiment-specific user-annotated training sets and experiment-specific user-defined segmentation parameters remains a major bottleneck for fully automating the analysis process. In this work we present LiveCellMiner, a highly flexible open-source software tool to automatically extract, analyze and visualize both aggregated and time-resolved image features with potential biological relevance. The software tool allows analysis across high-content data sets obtained in different platforms, in a quantitative and unbiased manner. As proof of principle application, we analyze here the dynamic chromatin and tubulin cytoskeleton features in human cells passing through mitosis highlighting the versatile and flexible potential of this tool set.

Using positional information to provide context for biological image analysis with MorphoGraphX 2.0.

Positional information is a central concept in developmental biology. In developing organs, positional information can be idealized as a local coordinate system that arises from morphogen gradients controlled by organizers at key locations. This offers a plausible mechanism for the integration of the molecular networks operating in individual cells into the spatially coordinated multicellular responses necessary for the organization of emergent forms. Understanding how positional cues guide morphogenesis requires the quantification of gene expression and growth dynamics in the context of their underlying coordinate systems. Here, we present recent advances in the MorphoGraphX software (Barbier de Reuille et al., 2015⁠) that implement a generalized framework to annotate developing organs with local coordinate systems. These coordinate systems introduce an organ-centric spatial context to microscopy data, allowing gene expression and growth to be quantified and compared in the context of the positional information thought to control them.

Quantitative Characterization of Macrophage, Lymphocyte, and Neutrophil Subtypes Within the Foreign Body Granuloma of Human Mesh Explants by 5-Marker Multiplex Fluorescence Microscopy.

Foreign bodies such as fibers of a surgical mesh induce a typical reaction with an inflammatory infiltrate that forms a surrounding granuloma. This infiltrate is dominated by macrophages, lymphocytes, and neutrophils, whereas its extent of collaboration is widely unknown. In this study, we analyzed 12 samples of surgical meshes explanted from humans by multiplex analyses with three different 5-marker panels - 1. macrophage panel: CD68, CD86, CD105, CD163, and CD206; 2. lymphocyte panel: CD3, CD4, CD8, CD20, and CD68; and 3. neutrophil panel: CD15, histone, MPO, NE, and CD68. Measurement of fluorescence intensity within nuclear masks resulting from DAPI nuclear staining allows exact quantification of cells considered "positive" at a user-defined mean intensity threshold of > 100. Obviously, however, there is no natural threshold as a biological criterion for an intensity that separates "positive" stained cells from unstained cells ("negative"). Multiplex staining of 5 markers always reveals a high rate of coexpression for almost all of the 25 possible marker combinations (= 32 combinations, when using 5 markers simultaneously). The present staining results demonstrate that various morphological and functional subtypes of macrophages, lymphocytes, and neutrophils are abundant in the foreign body granuloma (FBG), which were investigated by regions of interest (ROI) with an area of 1 mm2. The widespread coexpression of two or more markers underscores the complex collaboration network of the inflammatory infiltrate. The ability to combine spatial distribution with exact numerical analysis may offer new perspectives for our understanding of the complex interactions in this multidimensional process.

Quantitative mapping of keratin networks in 3D.

Mechanobiology requires precise quantitative information on processes taking place in specific 3D microenvironments. Connecting the abundance of microscopical, molecular, biochemical, and cell mechanical data with defined topologies has turned out to be extremely difficult. Establishing such structural and functional 3D maps needed for biophysical modeling is a particular challenge for the cytoskeleton, which consists of long and interwoven filamentous polymers coordinating subcellular processes and interactions of cells with their environment. To date, useful tools are available for the segmentation and modeling of actin filaments and microtubules but comprehensive tools for the mapping of intermediate filament organization are still lacking. In this work, we describe a workflow to model and examine the complete 3D arrangement of the keratin intermediate filament cytoskeleton in canine, murine, and human epithelial cells both, in vitro and in vivo. Numerical models are derived from confocal airyscan high-resolution 3D imaging of fluorescence-tagged keratin filaments. They are interrogated and annotated at different length scales using different modes of visualization including immersive virtual reality. In this way, information is provided on network organization at the subcellular level including mesh arrangement, density and isotropic configuration as well as details on filament morphology such as bundling, curvature, and orientation. We show that the comparison of these parameters helps to identify, in quantitative terms, similarities and differences of keratin network organization in epithelial cell types defining subcellular domains, notably basal, apical, lateral, and perinuclear systems. The described approach and the presented data are pivotal for generating mechanobiological models that can be experimentally tested.

3D fluorescence microscopy data synthesis for segmentation and benchmarking.

Automated image processing approaches are indispensable for many biomedical experiments and help to cope with the increasing amount of microscopy image data in a fast and reproducible way. Especially state-of-the-art deep learning-based approaches most often require large amounts of annotated training data to produce accurate and generalist outputs, but they are often compromised by the general lack of those annotated data sets. In this work, we propose how conditional generative adversarial networks can be utilized to generate realistic image data for 3D fluorescence microscopy from annotation masks of 3D cellular structures. In combination with mask simulation approaches, we demonstrate the generation of fully-annotated 3D microscopy data sets that we make publicly available for training or benchmarking. An additional positional conditioning of the cellular structures enables the reconstruction of position-dependent intensity characteristics and allows to generate image data of different quality levels. A patch-wise working principle and a subsequent full-size reassemble strategy is used to generate image data of arbitrary size and different organisms. We present this as a proof-of-concept for the automated generation of fully-annotated training data sets requiring only a minimum of manual interaction to alleviate the need of manual annotations.

Ventral striatal islands of Calleja neurons control grooming in mice.

The striatum comprises multiple subdivisions and neural circuits that differentially control motor output. The islands of Calleja (IC) contain clusters of densely packed granule cells situated in the ventral striatum, predominantly in the olfactory tubercle (OT). Characterized by expression of the D3 dopamine receptor, the IC are evolutionally conserved, but have undefined functions. Here, we show that optogenetic activation of OT D3 neurons robustly initiates self-grooming in mice while suppressing other ongoing behaviors. Conversely, optogenetic inhibition of these neurons halts ongoing grooming, and genetic ablation reduces spontaneous grooming. Furthermore, OT D3 neurons show increased activity before and during grooming and influence local striatal output via synaptic connections with neighboring OT neurons (primarily spiny projection neurons), whose firing rates display grooming-related modulation. Our study uncovers a new role of the ventral striatum's IC in regulating motor output and has important implications for the neural control of grooming.

Application of 3D MAPs pipeline identifies the morphological sequence chondrocytes undergo and the regulatory role of GDF5 in this process.

The activity of epiphyseal growth plates, which drives long bone elongation, depends on extensive changes in chondrocyte size and shape during differentiation. Here, we develop a pipeline called 3D Morphometric Analysis for Phenotypic significance (3D MAPs), which combines light-sheet microscopy, segmentation algorithms and 3D morphometric analysis to characterize morphogenetic cellular behaviors while maintaining the spatial context of the growth plate. Using 3D MAPs, we create a 3D image database of hundreds of thousands of chondrocytes. Analysis reveals broad repertoire of morphological changes, growth strategies and cell organizations during differentiation. Moreover, identifying a reduction in Smad 1/5/9 activity together with multiple abnormalities in cell growth, shape and organization provides an explanation for the shortening of Gdf5 KO tibias. Overall, our findings provide insight into the morphological sequence that chondrocytes undergo during differentiation and highlight the ability of 3D MAPs to uncover cellular mechanisms that may regulate this process.

Mechanical competition alters the cellular interpretation of an endogenous genetic program.

The intrinsic genetic program of a cell is not sufficient to explain all of the cell's activities. External mechanical stimuli are increasingly recognized as determinants of cell behavior. In the epithelial folding event that constitutes the beginning of gastrulation in Drosophila, the genetic program of the future mesoderm leads to the establishment of a contractile actomyosin network that triggers apical constriction of cells and thereby tissue folding. However, some cells do not constrict but instead stretch, even though they share the same genetic program as their constricting neighbors. We show here that tissue-wide interactions force these cells to expand even when an otherwise sufficient amount of apical, active actomyosin is present. Models based on contractile forces and linear stress-strain responses do not reproduce experimental observations, but simulations in which cells behave as ductile materials with nonlinear mechanical properties do. Our models show that this behavior is a general emergent property of actomyosin networks in a supracellular context, in accordance with our experimental observations of actin reorganization within stretching cells.

MondoA regulates gene expression in cholesterol biosynthesis-associated pathways required for zebrafish epiboly.

The glucose-sensing Mondo pathway regulates expression of metabolic genes in mammals. Here, we characterized its function in the zebrafish and revealed an unexpected role of this pathway in vertebrate embryonic development. We showed that knockdown of mondoa impaired the early morphogenetic movement of epiboly in zebrafish embryos and caused microtubule defects. Expression of genes in the terpenoid backbone and sterol biosynthesis pathways upstream of pregnenolone synthesis was coordinately downregulated in these embryos, including the most downregulated gene nsdhl. Loss of Nsdhl function likewise impaired epiboly, similar to MondoA loss of function. Both epiboly and microtubule defects were partially restored by pregnenolone treatment. Maternal-zygotic mutants of mondoa showed perturbed epiboly with low penetrance and compensatory changes in the expression of terpenoid/sterol/steroid metabolism genes. Collectively, our results show a novel role for MondoA in the regulation of early vertebrate development, connecting glucose, cholesterol and steroid hormone metabolism with early embryonic cell movements.

In most animals, a protein called MondoA closely monitors the amount of glucose in the body, as this type of sugar is the fuel required for many life processes. Glucose levels also act as a proxy for the availability of other important nutrients. Once MondoA has detected glucose molecules, it turns genetic programmes on and off depending on the needs of the cell. So far, these mechanisms have mainly been studied in adult cells. However, recent studies have shown that proteins that monitor nutrient availability, and their associated pathways, can control early development. MondoA had not been studied in this context before, so Weger et al. decided to investigate its role in embryonic development. The experiments used embryos from zebrafish, a small freshwater fish whose early development is easily monitored and manipulated in the laboratory. Inhibiting production of the MondoA protein in zebrafish embryos prevented them from maturing any further, stopping their development at an early key stage. This block was caused by defects in microtubules, the tubular molecules that act like a microscopic skeleton to provide structural support for cells and guide transport of cell components. In addition, the pathway involved in the production of cholesterol and cholesterol-based hormones was far less active in embryos lacking MondoA. Treating MondoA-deficient embryos with one of these hormones corrected the microtubule defects and let the embryos progress to more advanced stages of development. These results reveal that, during development, the glucose sensor MondoA also controls pathways involved in the creation of cholesterol and associated hormones. These new insights into the metabolic regulation of development could help to understand certain human conditions; for example, certain patients with defective cholesterol pathway genes also show developmental perturbations. In addition, the work highlights a biological link between cholesterol production and cellular responses to glucose, which Weger et al. hope could one day help to identify new cholesterol-lowering drugs.